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The On-Off problem, aka. Li-Ma problem, is a statistical problem where a measured rate is 
the sum of two parts. The first is due to a signal and the second due to a background, both of 
which are unknown. Mostly frequentist solutions are being used that are only adequate for high 
count numbers. When the events are rare such an approximation is not good enough. Indeed, in 
high-energy astrophysics this is often the rule rather than the exception. 

I will present a universal objective Bayesian solution that depends only on the initial three param¬ 
eters of the On-Off problem: the number of events in the “on” region, the number of events in the 
“off” region, and their ratio-of-exposure. 

With a two-step approach it is possible to infer the signal’s signihcance, strength, uncertainty or 
upper limit in a unihed a way. The approach is valid without restrictions for any count number 
including zero and may be widely applied in particle physics, cosmic-ray physics and high-energy 
astrophysics. I apply the method to Gamma Ray Burst data. 
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1. Introduction 


Typical counting experiments measure discrete sets of events. Such data are often [1, 2] mod¬ 
eled with the Poisson distribution. The Poisson distribution may be approximated by a normal 
distribution when measuring many events. However, when data is rare, such an approximation is 
not good enough. In Fig. 1, a typical example of a low count data sample is shown. In this case 
it is an observation of a Gamma Ray Burst (GRB) with the Fermi-LAT instrument. The question 
arises: What do you do when it simply impossible to “go out and get more data”? 



Time since GBM Trigger (s) 


Figure 1: A typical low count high-energy astrophysics data set. It shows gamma rays measured from 
GRB080825C as observed by Fermi-LAT. Black dots represent the energy measurement, the gray bars rep¬ 
resent the number of photons. Figure reproduced from [3] 


2. The On-Off Problem 

In the On-Off problem, also known as Li-Ma problem, one would like to infer a signal rate 
in the presence of an imprecisely known background rate. The measurement consists of the ob¬ 
servation of Non events in some “on” region with a potential signal and Aoff events in some “off” 
region, known to be signal free. Additionally to the number counts in the on-and off regions, there 
is a third parameter. This parameter is the ratio a of exposures for the “on” and “off” regions and 
taken to be known with negligible uncertainty. In the case of gamma ray astronomy, Berge et al. [4] 
explain the problem and its parameters well. 

The common frequentist analyses, based on likelihood ratios and other methods [1, 2, 5] often 
assume normal distributed random numbers and therefore lose their foundation when applying 
them to low count numbers. They also get into trouble at the border of the physical parameter 
space [5]. 

The common Bayesian solutions to the On-Off problem are either subjective Bayesian (us¬ 
ing proper posteriors [6]) or they avoid specifying the alternative hypothesis at all by using some 
tail-area probability inference in the spirit of p-values [7]. Subjective Bayesian methods usually in¬ 
troduce a fourth, subjective, parameter (often the upper limit to the signal parameter) which makes 
the probability statement somewhat dependent on the individual physicist. The tail-area methods, 
besides ignoring the beauty of Bayesian hypothesis testing, seem to overestimate the probabil¬ 
ity [6]. 

Most of all none of the methods in the literature so far cover the full range of the problem: 
First, the probability that the observed counts are due to background only has to be calculated. 
Second, the signal contribution has to be estimated. In this proceeding I present a two-step objective 
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Bayesian solution to the full On-Off problem [8], inspired by [9], that addresses these issues in a 
unified a way. 

3. Methods Development 

The idea behind objective Bayesian analysis simple. One takes, in a sense, “flat” priors rep¬ 
resenting the lack of knowledge. These are usually improper (do not integrate to one). However, 
in combination with Bayes theorem these can be used to produce proper posteriors and answer a 
basic what-if question: What is the result if the data were dominant? 

One particularly popular objective Bayesian prior is Jeffreys’s rule [10, 11]. Jeffreys was 
motivated by invariance requirements and suggested to take a specific objective prior to make the 
result (the posterior) invariant under re-parametrization. This prior is a keystone in this analysis. 

The analysis follows the method outlined by [9] and is done in two steps. First, the odds 
that the observed counts are due to the background model are calculated. If this is smaller than a 
previously defined value, the signal is said to be detected. Second, the signal contribution or upper 
limit is calculated, depending on whether the detection limit has been reached. The first is done 
with objective Bayesian hypothesis testing via Bayes factors. The second is done with objective 
Bayesian estimation. 

3.1 First step: Hypothesis testing 

One problem that appears is that objective priors are only defined up to a proportionality 
constant and those constants become relevant in this case. A full discussion of the topic can be 
found in [8, 12]. In short, there is no generally agreed objective Bayesian hypothesis testing. I 
suggest to use a method sometimes called “minimal sample device” [13, 14], in order to fix the 
issue with the proportionality constants. The evaluation of these assumptions in the case of the 
On-Off problem can be found in [8, 12]. After all, the odds of the background model over the 
signal model are 


where y and 5 are defined using the Gamma function r(x) and the hypergeometric function 
2 F 1 {a,b-,c-,z)- 



(3.2) 


A signal detection based on Eqn. 3.1 may be claimed when the resulting odds of the background 
model are low. I propose to use use a ’’Bayesian z-value“, similar to [7] 


5b = V2err‘(l-Boi) 


(3.3) 
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where Bqi = 5.7 x 10^^ would correspond to S'b = 5 or ”5 sigma“. This definition allows for an 
easy comparison with frequentist significance methods [ 8 , 12]. However one must keep in mind 
that the odds of a model and the frequency of an outcome are two different things. Bqi explicitly 
weighs alternative models, while the frequentist methods do not. 


3.2 Second Step: Signal Estimation 

After determining the Bayes factor of the background model over the signal model, one pro¬ 
ceeds to estimating the signal contribution. This is done via objective Bayesian estimation. If the 
data show a significant detection the signal model can be assumed to be true. Then, the most prob¬ 
able signal parameter value should be calculated and a physical error interval should be given. If 
the data show no significant detection, an upper limit on the signal parameter should be calculated, 
assuming that the signal is there (i.e. the signal model is true) but too weak to be measured. In both 
cases one needs the conditional probability P(As|Aon,Aoff,7/i) of the signal As, given the number 
counts and the signal model H\ . The improper prior is acceptable in this case because the propor¬ 
tionality constant c\ cancels and the posterior is proper. After marginalization over the background 
parameter Abg, the result is (calculation in [ 8 ]) 


P(As|A^on,fVoff,/7l) 


Pp(fVon+fVoff|As) 


u [; +fVoff, 1 +fVoff+ fVon, (1 + As] 

2 F 1 +77off, 1 +NoS + Non, I +77off; —’ 


(3.4) 


as expressed in terms of three functions, namely the Poisson distribution Pp(A|A), the regular¬ 
ized hypergeometric function 2 F 1 {a,b',c',z) = and the Tricomi confluent hypergeomet¬ 

ric function \J{a,b,z)- This posterior contains the full signal parameter information. In order 
to state a flux, one should take the mode A*, of the posterior distribution P (As|/7on)A^off)^i)> as 
signal estimator. The error on the signal estimator can be evaluated numerically from the cumu¬ 
lative distribution function. On interesting choice is the highest posterior density interval (HPD) 
[■^min, Amax] [ 8 ] Containing 68 % probability, calculated as 



P(As|fVon,fVoff,7/i)r/As = 0.68, 


(3.5) 


together with the constraint 


P(Aniin|fVon,fVoff,/7l) = P(A^ax|fVon,fVoff,7/l) . (3.6) 

In case an upper limit should be calculated one can solve the cumulative distribution function, 
for instance, for a 99% probability limit A 99 on the signal parameter As as 

/ P(As|iVon,fVoff,/7i)^/As = 0.99. (3.7) 

Jo 

These results are natural in a Bayesian approach of the problem but hard to calculate in a frequen¬ 
tist approach. Most frequentist methods particulaidy struggle with the mai'ginalization and fail at 
the border of the parameter space. In this approach, all possible number counts are dealt with in a 
uniform way, no matter if zero counts or thousands of counts. A further benefit is that the signal pos¬ 
terior probability intervals are always physically meaningful (i.e. positive A^*, A„,,,), Amax, A 99 ,...). 
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As an application of the developed method, I want to demonstrate the calculations on two 
measurements of gamma rays from GRBs. These are extraterrestrial flashes of gamma rays mostly 
lasting only a few seconds. One interesting question is how exactly GRBs produce high energy 
gamma rays [3]. Because of the GRBs duration and fluence, satellites and Cherenkov telescopes 
measure only few events during the flare itself or shortly after. 

The first example is the GRB080825C as seen by Fermi-LAT (see Sec. 1). In Fig. 1 one sees 
the light curve in gamma rays > 80MeV. In total there were Non = 15 on events and Aoff = 19 off 
events, with an exposure ratio of a = 33/525 [3]. Running the numbers shows that the odds of 
the background model over the signal model are (Eqn. 3.1) Bq\ = 9.66 x 10^^°, or as expressed 
with the nonlinear scale of Eqn. 3.3, Sb = 6.11. These numbers compare well to the published 
value of Sii-ma = 6.4 [3]. Clearly, the odds of the background model are low and the GRB is 
therefore detected. Then, in the second step, one performs the signal estimation. The result is 
plotted in Eig. 2. The result of As = 13.28/)3'49 is in good agreement with the published reference 



3.S 

Figure 2: The conditional probability f’(As|Aon,A(,ff,//i) of the signal As, given the Fermi-LAT number 
counts of GRB080825C. The blue band indicates the HPD interval for the signal parameter posterior proba¬ 
bility. Figure reproduced from [8]. 

of ARef. = 13.7. 

The second example is the GRB080330 as observed by the VERITAS Cherenkov telescope [15]. 
The measurement shows Non = 0 on events and Voff = 15 off events, with an exposure ratio of 
a = 0.123 [8]. The corresponding odds of the background model are Rot = 2.29, unsurprisingly 
favoring the null hypothesis as not a single on event was detected. Now, assuming that the source is 
there one can put an upper limit to Aj. The result is plotted in Eig. 3. The published value uses a fre- 
quentist upper limit setting method, popularized by Rolke et al. [5]. Their result is = 2.4 [ 8 ], 
which somewhat lower than the number from Eqn. 3.7, A 99 = 4.10. A detailed analysis indicates [ 8 ] 
that, especially at the border of the parameter space for Non < otNoft, Rolke’s method is an overes¬ 
timation and therefore limited. These limits are overcome by the Bayesian method. 

4. Validation and Discussion 

In order to validate the method and to check if the assumptions made are sensible, an extensive 
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Figure 3: The conditional probability P(As|A^on,A^off,^/^i) of the signal As, given the VERITAS number 
counts of GRB080330. The blue band indicates the 99% probability upper limit to the signal parameter. 
Figure reproduced from [8]. 


validation was made [8]. The validation shows that the two-step method behaves well in all test- 
case examples, in particular at Von ~ ctVoff. The objective Bayesian hypothesis testing converges to 
the results from other methods for high count numbers. The objective Bayesian signal estimation 
can reconstruct the true signal parameter As with a good error estimate. 


5. Conclusion 

Claiming detections, setting credibility intervals, or setting upper limits can be unified over 
the whole On-Off problem parameter range in one consistent two-step objective Bayesian method. 
An example implementation in Python can be downloaded from the public git-repository 
https://bitbucket.org/mknoetig/obayes_onoff_problem. 


References 

[1] T. P. Li and Y. Q. Ma, Analysis methods for results in gamma-ray astronomy, Astrophys. J. 272 (1983) 
317-324. 

[2] R. D. Cousins, J. T. Linnemann, and J. Tucker, Evaluation of three methods for calculating statistical 
significance when incorporating a systematic uncertainty into a test of the background-only 
hypothesis for a poisson process, Nucl. Instrum. Methods A 595 (2008), no. 2 480-501. 

[3] Fermi LAT/GBM Collaborations Collaboration, A. Abdo et ah, Fermi observations of high-energy 
gamma-ray emission from grb 080825c, Astrophys. J. 707 (2009), no. 1 580. 

[4] D. Berge, S. Funk, and J. Hinton, Background modelling in very-high-energy y-ray astronomy, 

Astron. Astrophys. 466 (May, 2007) 1219-1229. 

[5] W. A. Rolke, A. M. Lopez, and J. Conrad, Limits and confidence intervals in the presence of nuisance 
parameters, Nucl. Instrum. Methods A 551 (2005), no. 2 493-503. 


6 



















Objective Bayesian On-Off Analysis 


Max Ludwig Ahnen 


[6] P. Gregory, Bayesian logical data analysis for the physical sciences. Cambridge University Press, 
2005. 

[7] S. Gillessen and H. L. Harney, Significance in gamma-ray astronomy - the Li - Ma problem in 
Bayesian statistics, Astron. Astrophys. 430 (2005) 355-362. 

[8] M. L. Knoetig, Signal discovery, limits, and uncertainties with sparse on/off measurements: an 
objective bayesian analysis, Astrophys. J. 790 (2014), no. 2 106. 

[9] A. Caldwell and K. Kroninger, Signal discovery in sparse spectra: A bayesian analysis, Phys. Rev. D 
74 (2006), no. 9 092003. 

[10] H. Jeffreys, Theory of probability. Clarendon Press Oxford, 1961. 

[11] Particle Data Group Collaboration, J. Beringer et al.. Review of particle physics, Phys. Rev. D 86 
(2012)010001. 

[12] M. L. Ahnen, On the on/off problem, in Bayes Forum Munich, 2014. 

[13] D. J. Spiegelhalter and A. F. M. Smith, Bayes factors for linear and log-linear models with vague 
prior information, J. R. Statist. Soc. B 44 (1981), no. 3 377-387. 

[14] J. K. Ghosh and T. Samanta, Nonsubjective bayes testing — an overview, J. Statist. Plann. Inference 
103 (2002) 205-223. 

[15] VERITAS Collaboration Collaboration, V. A. Acciari et al., VERITAS Observations of Gamma-Ray 
Bursts Detected by Swift, Astrophys. J. 743 (2011) 62. 


7 



